Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed Speculative Generator #75

Merged
merged 1 commit into from
Sep 30, 2023
Merged

Fixed Speculative Generator #75

merged 1 commit into from
Sep 30, 2023

Conversation

SinanAkkoyun
Copy link
Contributor

Hey! Just wanted to test the speculative.py example and noticed that the speculative generator did not use the (I suppose new) sampler syntax, so this is the fix for that. ^^

# from
token, prob = ExLlamaV2Sampler.sample(logits, draft_settings, past, randoms[i])

# to:
token, prob, _ = ExLlamaV2Sampler.sample(logits, draft_settings, past, randoms[i], self.tokenizer)

# etc :)

@turboderp
Copy link
Owner

Yeah, for token healing the sampler needs to call the tokenizer. So I missed that.

@turboderp turboderp merged commit 90d29b0 into turboderp:master Sep 30, 2023
@SinanAkkoyun
Copy link
Contributor Author

I see, I'm so glad that you implemented speculative sampling as a nice generator, thank you!

@SinanAkkoyun SinanAkkoyun deleted the fix-speculative-gen branch October 2, 2023 18:36
anchortense pushed a commit to anchortense/exllamav2-logit-threshold-samplers that referenced this pull request Oct 21, 2024
@turboderp
Merge pull request turboderp#75 from SinanAkkoyun/fix-speculative-gen
aa64f81 
Fixed Speculative Generator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@SinanAkkoyun @turboderp